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We study spin susceptibility and magnetic order at the edges/surfaces of two-dimensional and 
three-dimensional topological insulators when the Fermi surface is nested. We find that due to spin- 
momentum locking as well as time-reversal symmetry, spin susceptibility at the nesting wavevector 
has a strong helical feature. It follows then, a helical spin density wave (SDW) state emerges 
at low temperature due to Fermi surface nesting. The helical feature of spin susceptibility also 
has profound impact on the magnetic order in magnetically doped surface of three dimensional 
topological insulators. In such system, from the mean field Zener theory, we predict a helical 
magnetic order. 
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I. INTRODUCTION 

In the past few years, a new family of materials 
called topological insulators (TIs) have been theoreti- 
cally predicted^"— and then experimentally observed^"— 
A TI has a energy gap in the bulk and gapless excita- 
tion in the edge/surface, which is due to the nontrivial 
band topology and protected by time reversal symmetry. 
TIs possess a nontrivial topological order, which distin- 
guishes them from simple band insulators. In a two di- 
mensional (2D) TI, which is also known as a quantum 
spin Hall insulator, the edge states form a helical Lut- 
tinger liquid. The surface states of three dimensional 
(3D) TIs form a "helical metal" with Dirac cone like 



spectrum; 



3.4.8.9.12-14 



(Specifically, in this paper, we are 



interested in a class of 3D TIs where the surface state 
consists of a single Dirac cone. Examples are Bi2Te3 )^°i^^ 
BiaSeg^'i^ii^ SbaTegji^'i^ TlBiSea^iii^, and many other 
material o^^i^° . Due to large bandgap and high purity, 
these materials have great potential for application and 
scientific research. ^^). An important feature is that spin 
and momentum are closely correlated in the edge / surface 
states of TIs, which leads to many unusual effects2i"— 
and potential applications in spintronics and quantum 
computation.^^ 

Recently, it was found in Bi2Te3 that the as Fermi en- 
ergy increases from the Dirac point, the shape of Fermi 
surface gradually changes from a circle, first to a hexag- 
onal shape, and then to a snowflake-likc i This phe- 
nomenon was also found in other 3D TIs with similar 
structuresji^ii^^"— This kind of band structure is the- 
oretically reproduced by Fu from the k ■ p theoryi^ For 
a certain range of energies, the Fermi surface is almost a 
hexagon, which leads to strong nesting at three wavevec- 
tors and possible instability to the formation of SDW 
states. 

In this paper, we study spin susceptibility and mag- 
netic order at the edges/surfaces of TIs when the Fermi 
surface is nested. We find that due to the one-to-one cor- 



respondence between spin state and momentum ("spin- 
momentum locking" ) as well as time reversal symmetry, 
the spin susceptibility function at nesting wavevector has 
a strong helical feature. It follows then, a helical SDW 
state emerges at low temperature due to Fermi surface 
nesting. We present a mean field theory of the helical 
SDW state. The helical feature of the spin susceptibility 
function also has profound impact on the magnetic order 
in the magnetically doped surfaces of 3D TIs. In such 
system, from the mean field Zener theory, we predict a 
helical magnetic order. 



II. HAMILTONIAN, SPECTRUM AND 
EIGENSTATES 

We consider the situation where Fermi surface exhibits 
strong nesting feature. Examples are, the edge states in 
a 2D TI, and the surface state in the 3D TI Bi2Te3 with 
Fermi energy in the range of [0.13, 0.23] eV where the 
Fermi surface is almost a hexagonJ^i [see Fig. 1]. The 
hexagonal shape of Fermi surface also exists in the sur- 
face states of many other 3D TIs, such as Bi2Se3fi^i^ 
Sb2Te3,-21'28 TlBiSe^ii^, TlBiTea^a and a recently dis- 
covered large class of 3D TIs in Ref. [lO. In the following 
we would call these materials the "Bi2Se3 class". 

The Hamiltonian of the edge states in a 2D TI can be 
written as^^ 



Ho 



kafi 



(1) 



where vq is the Fermi velocity. The spin orientations of 
the eigenstates, through proper choosing of the spin co- 
ordinates, have been taken as up and down. The eigenen- 
ergies and eigenstates are 



e±{k) = ±vo\k\, u±{k) 



( ^ (2) 



where 8 is the Hcaviside function. The nesting vector is 
Q = 2kF. 
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FIG. 1. (Color online) (a) Spectrum of the edge states of 2D 
TIs as well as schematic of the nesting wave- vector Q = 2kF- 
Nesting wavevector connects spin-down and spin-up states, 
(b) Energy contour of the surface states of 3D TIs for three 
typical energy, 0.2,0.7,2 Es, as well as schematic of nesting 
wave-vectors for hexagonal Fermi surface. The numbers label 
the Fermi Arcs. There are three nesting wavevectors, Qi from 
Arc 4 to Arc 1, Q2 from Arc 6 to Arc 3, Q3 from Arc 2 to 
Arc 5. 



Obviously, the physical properties are universal in these 
materials, with exact quantitative scaling by properly re- 
covering the dimension via Eg and ks within the above 
model. The parameters, v, 7, Eg and ks of several 3D 
TIs inferred from experiments are listed in Table I. 



TABLE I. Hamiltonian parameters of the surface states in 
3D TIs with hexagonal symmetry. (" from Ref. l27l. ^ from 
Ref.[i3, " inferred from Ref.[ii, inferred from RefllS.) 



V (eVA) 7 (eVA3) E, (eV) k, (A"^) 



BizTeg": 


2.55 


250 


0.26 


0.1 




3.55 


128 


0.59 


0.17 


TlBiSea": 


3.1 


182 


0.4 


0.13 


GeBi2Te4'*: 


2.37 


99 


0.37 


0.15 


BiaTeaSe'*: 


7.25 


580 


0.81 


0.11 



Throughout this paper, we focus on the situation 
where the Fermi energy Ep is high and the temperature 
is low, so that states far below the Fermi surface (such as 
those below the Dirac point) is irrelevant. This regime is 
easy to achieve in the Bi2Se3 class of 3D TIs, thanks to 
the large bandgap and Dirac velocity. 

The Hamiltonian for the surface states in 3D TI with 
a single Dirac cone, is given in Refs. 2Ii and S3- Keep- 
ing only the dominant terms, the Hamiltonian is (with 
X and y axes along the F — A' and F — M directions 
respectively) ^L2^ 

Ho = v{k.^ay~kyaa,) + ^{kl + kl)a,, (3) 

where k± = k^ i iky. The eigenenergy and spin states 
are2^ 

e±(k) = ±v/w2fc2 +72fc6 cos2 36*^,., (4) 
, , _ I ( vk{s\n6k +icos6'fe) \ , 
" IJW) \ ^± (1^) - 7fc=^ cos 3^? J ' 

with k fc(cos6'fc,sin6'fc), A(k) = w^fc^ -|- (e+(k) - 
^k^ cosWk)^ . The Hamiltonian is time-reversal invari- 
ant. The last term reduces the symmetry from Coov to 
C^v . The Hamiltonian describes the surface state of the 
Bi2Se3 class of 3D TIs.— »^ It is instructive to rescale the 
Hamiltonian withf^l 

Ho = EsH'o, k = ksk', (6) 

where 

Es = vy/vpf, ks = \fvpi. (7) 
Then the Hamiltonian becomes 



If the Fermi energy is in the range of Q.bEs < Ep < 
0.9Es, the Fermi surface is almost a hexagon, which ex- 
hibits strong nesting feature. There are three nesting 
wave- vectors 

Qi = 2fco(l,0), Q2 = 2fco(-^,^), 

Q3 = 2fco(-^,-^), (9) 

where fco is determined by yVfc^+T^fc^ — Ep. 

At higher Fermi energy, the Fermi surface is dis- 
torted to snowflake-like and new nesting wave-vectors 
emerges. ^^'^^ However, at such high Fermi energy usu- 
ally the bulk conduction band is also occupied, which 
complicates the situation and is not interested in this 
paper. 

The electron-electron interaction consists of the long- 
range Coulomb interaction and the short-range Hubbard 
interaction. The former does not affect the spin suscepti- 
bility [to the random phase approximation (RPA)^] and 
is hence ignored. The onsite Hubbard interaction is writ- 
ten as 

(10) 

q 

where cr(q) = J2kafi '^a'^ajSCk+qis with cr being Pauh 
matrices vector and a, /3 being spin indices. For positive 
Hubbard U (repulsive interaction), Fermi surface nest- 
ing leads to the SDW instability and transition into the 
SDW state at sufficient low temperature. Whereas, for 
negative U (attractive interaction) , it leads to the charge 
density wave (CDW) instability and the emergence of 
CDW state. Here we assume, as in most cases, U > 0. 
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III. SPIN SUSCEPTIBILITY AND SDW 
INSTABILITY 

A. Edge states of 2D TIs 

The linear spin susceptibility function is 

/■oo 

xMq,io)^i di([a^(g,t),r7.(-g,0)])e'*('^+'O^), (11) 



where n^v = {x,y,z) or (±,z) with a± — \{<yx ± i^j^)- 
In the edge of a 2D TI, at the nesting vector Q = 2kp^ 

the only nonzero terms are xh {—Q,^) and x hlQi"^) 

as the nesting wavevector connects spin-down and spin- 
up states. This helical feature of the spin susceptibility 
function is due to spin-momentum locking as well as time 
reversal symmetry. The two susceptibility function are 

actually related by [x^ (— Q,— cj)]* = x h{Qi^)- In 

the absence of interaction, the spin susceptibility is 



(12) 



where ^ki = £ki — Ep and np is the Fermi distribution. 
Including the Hubbard interaction within RPA, one gets 



x°-+(Q,^) 
i-Ux'l+iQ^uj)' 



(13) 



Due to Fermi surface nesting, the spin susceptibility 
X°^(Q,0) is divergent at low temperature. Actually, di- 



rectly from Eq. ([T2|) . one can show that X-+(Q:0) ~ 
-^log{Ep/kBT). That is, the spin susceptibility func- 
tion is logarithmically divergent with decreasing temper- 
ature. This signals the SDW instability. The feature that 
only X-+(Q7 0) diverges indicates a helical SDW order. 

The above treatment based on the Fermi-liquid theory 
is of course invalid for one-dimensional electron system, 
but it sheds some light on the problem. In the following, 
we analyze the problem via the bosonization theory. 

Following Wu et al.,ii the bosonized Hamiltonian in 
the presence of Umklapp scattering can be expressed as 



H 



1 

2^ 



dx 
9u 



uK{we{x)y - 

-cos(4(/)(x)). 



2(7ra)2 

where the bosonized fermion fields are 



(14) 



ict>R{x) 



-ikpx 



V27ra 

(15) 

with (j) = ((/lij-f (/)l)/2, 9 = {(j)R-(l)L)/'^- K the Luttinger 
parameter, u is the renormalized Fermi velocity. Qu is 
the Umklapp scattering strength, a is a short-distance 
cutoff. Due to symmetry reasons, the only possible in- 
stabilities are SDW and singlet superconductivity (SC). 
This is because CDW and triplet SC instabilities pair 



particles with the same spin, i.e., terms like ^'It^ipL^ for 

CDW and for triplet SC, are impossible. The 

bosonized form of spin operators are 



a+{x) = 'tpl^^{x)'ipLi{x) 
(j_{x) = 'iIjI^{x)iIjr^{x) 



— i2kFX 

27Ta 



27ra 



(16) 



From standard bosonization theory, '^'^ the Umklapp term 
becomes relevant when K < 1/2. Then RG flow will go 
to a strong coupling fixed point, gu oo, and the (j) field 
will become ordered. Depending on the sign of gu, the 
ordered value of 6 is 



TT 2mr 

4 + — , 5u > 0, 

, 2m: 

(0)=O + — , 5„<0. 



(17) 



This signifies a true phase transition. Then the spin op- 
erators, which have zero expectation values in the non 
ordered phase, also acquire nonzero average value across 
the transition. 



CTx = 0-4 



[(Ty) = -{(Tj, 



(27ra)2 
-2 



(2™)' 



cos{2kpx — ((/))), 
■sin(2fcFa;- {(j))), (18) 



which shows helical structure and is consistent with the 
mean field result. We note that very recently a similar 
calculation has been carried out by Kharitonov,'^^ con- 
sidering helical Luttinger liquid in the proximity to a 
ferromagnet, which also agrees with our result. 



B. Surface states of 3D TIs 

a. General considerations On the surface of a 3D 
TI, the nesting vector will connect states which are not 
Kramers pairs and hence their spin states are not antipar- 
allel. As a consequence, the spin susceptibility is finite 
in all directions. The free spin susceptibility function in 
this case is 

x^,.(Q, =2^ 7 . -0+ 9A^, Q)9AK Q) 



where 



5^(k,Q) = (u+(k)|aJu+(k + Q)) 



(19) 



(20) 



with fi,iy = {x,y,z). One can note that, 
5p(k, Q)(7*(k, Q) is a bilinear tensor, where f;p(k, Q) is 
its eigenvector with eigen-value 



^,(k,Q)=El.9^(k,Q)|^ 



(21) 
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Moreover, if 
and 

Xc(Q,o) = 



5o(k,Q) = (u+(k)|u+(k + Q)), 



V a+-ek+Q+ + *o 



(22) 



|.go(k,Q)|2, (23) 



still has strong helical feature. Specifically, one of the 
eigenvalues of the spin susceptibility tensor X°i/(Qij 0) is 
much larger than the other two [Fig. 2(b)], which corre- 
sponds to a spin density response with helicity very close 
to unity [Fig. 2(c)]. The helical spin rotating axis is indi- 
cated in Fig. 2(a) as rig, which lies in the y-z plane with 
an angle 0„ [Fig. 2(c)] between y-axis. 



is the charge susceptibility, then 

|5o|' + Ks(k,Q) = 2, 1 < «:<,(k,Q) < 2. 



(24) 



and a "complementary relation^^ between the spin- and 
charge- susceptibility, 



Exmm(Q.o) + Xc(Q,o) = x"(Q,o) 



(25) 



with x°(Q,0) ^ Ek "lt-g:iq;:for\ due to spin- 
momentum locking. We then introduce the spin density 
operator 



ag(k,Q) 



1 



Kg(k, Q) 



EffM(k,Q)4a<Ck+Q/3. (26) 



pi,al3 



One can show that 

(u+(k)|ag(k,Q)|w+(k-KQ)) = 1. 
And, any spin density operator 



(27) 



a/(k, Q) = E /;(k)cL<^MCk+Q/3, (28) 

^l,,a|3 

which is perpendicular to Ug, i.e., ^ g*^{^) f — 0, 
has 



(u+(k)|a/(k,Q)|7.+ (k-HQ)) =0. 



(29) 



Therefore, only one spin density susceptibility is nonzero 
(if only k and k + Q states are concerned) , which is de- 
fined by crg(k, Q). This intriguing feature is due to spin- 
momentum locking. 

h. Helical feature Consider, if k and k -|- Q are time 
reversal pair states, of which spin orientations are op- 
posite, the spin density operator i7g(k, Q) is helical, as 
we learn from the case in the edges of 2D TIs. Unfortu- 
nately, here (;^(k, Q) is k dependent. Besides, only for 
some very special k states, such as kg — (— fco,0) and 
ko + Qi = (fco, 0), the two states are time reversal pairs. 
If the contributions of all k states are summed, the helical 
feature may be smeared out. 

To study this case, we calculate the free spin sus- 
ceptibility function xJliy(Qi,0) numerically for Ep £ 
[0.5, 0.9] Es where the Fermi surface is hexagonal (Qi = 
(2fco, 0) [see Eq. ([9])] connects Fermi Arc 4 and Arc 1 [see 
Fig. 1(b)]). Our results are presented in Fig. 2. From the 
calculation, we find that, though the exact helical spin 
susceptibility is ruled out, the spin susceptibility function 





0.5 0.7 

(b) Ep ( E, ) 



.i 0.99 
0.98 



0.5 0.7 0.9 

W E,.(EJ 

FIG. 2. (Color online) (a) Schematic of the spin density re- 
sponse corresponding to the largest eigen-value of the spin 
susceptibihty tensor x''(Qii 0). is the helical rotating axis. 
The blue arrows indicate spins rotating around the ng a:xis. 
rig lies in the y-z plane with an angle On between y axis, (b) 
Eigenvalues (positive definite) of the tensor X°(Qii 0) as func- 
tion of Fermi energy (in unit of Es). (c) The hehcity and the 
angle On as function of Fermi energy. All data are calculated 
at ksT = 1.3 X lO'^Ss. 

c. Physical explanations To understand the above 
results, let's consider the problem that how the spin sus- 
ceptibility function gets maximized at Qi. The spin sus- 
ceptibility function for an arbitrary spin density opera- 
tor, a/(Qi) = E^/;f^M(Qi) I/mP = 1 (normalized), 

anda^Qi)-^- 



k.a/jCkQ^r^'^Ck+Qi/jJ, IS 



x/(Qi,o) 



k 



k+Qi- 



Ck+Qi 



^Kg(k,Qi) 



«g(k, Ql), 



(30) 



The spin susceptibility gets contribution mainly from the 
states satisfying the nesting condition and from the vicin- 
ity. The above equation also indicates that the contribu- 
tion is proportional to the factor Kg(k, Qi). It is easy to 
show that for the special wavevector kg = (— fco,0), the 
nesting condition is exactly satisfied, fko+ = '?ko+Qi+ = 
0. Moreover, from (denote (7° = 5^t(ko, Qi)), 



= 9l 



'Vko 



(31) 

one finds that the magnitude Kg(k, Qi) is largest at kg = 
{—ko, 0) with Kg(ko, Qi) = 2. As ko is at the center of the 



5 



Fermi Arc 4, where the main contribution to spin suscep- 



tibihty comes, the overlapping factor 



y f* 



^Kg(k,Qi 



is also very large, if /* = g^* / y^KgCko, Qi). Calculation 
indicates that the overlapping factor is very close to unity 
in the vicinity of Fermi Arc 4. Therefore, the spin density 
operator cr/ which maximizes the spin susceptibility func- 

tion should be a°{Qi) = ff°*'^/.(Qi)/ V«9(ko, Qi)- 
This observation is very close to the truth, except 
that it ignores some delicate part. Although the fac- 

' -F(Ck+)-"F(^k+Qi + ) 



is peaked at ko 



-A:o,0) (as 



tor ^ ^ 

^ko+ — Cko+Qi+ — 0), the dispersion, however, is asym- 
metric around ko along the direction, as the spectrum 
is nonlinear [see Eq. Q and Fig. 1]. Therefore, the spin 
density operator which maximize the spin susceptibility, 
denoted as ctg, deviates slightly from ct°. Nevertheless, 
ac is very close to cr^ and keeps most of the features of 
CTg, especially the helical feature. This feature even per- 
sists to room temperature, thanks to the large bandgap 
and Dirac velocity in the Bi2Se3 class of 3D TIs. 
One can write aciQi) = G'*^cr^(Qi), where 



E 



= 1 



(32) 



From Eq. pip , we know that Gix is a pure imaginary 
number, whereas Giy and Giz are real numbers. Also, 



is 



rig = (0, cos 6'„ , sin 6*,, 



(33) 



with 9n — Arcta,n{Giz/Giy) — 7r/2, agreeing with the re- 



sults in Fig. 2. The helicity is ^^Gjy + GjJGi^l. From 

Eqs. p2p and pip , one can see that the helicity is indeed 
close to unity. 

d. Systematic results In Fig. 3(a), we plot the 
largest eigenvalue of the spin susceptibility tensor, 
Xg(Q, 0), as function of |Q| with Q along the x-direction. 
It is seen that the spin susceptibility function has a strong 
peak at the nesting wave- vector Qi, especially when 
Ep = Q.lEg where the Fermi surface is almost a perfect 
hexagon. We also plot the spin susceptibility function 
at a higher temperature T = 100 K (blue dotted curve) 
with chemical potential 0.7 Eg. We find that the nesting 
and the helical features persist to high temperature. And 
the largest eigenvalue of the spin susceptibility tensor is 
still much larger than the other two for Q ^ Qi, at high 
temperature. 

We present a two-dimensional plot of Xg(Q,0) in 
Fig. 3(b). It is seen that the spin susceptibility function 
peaks at regions close to the nesting wavevectors ±Qj 
i = (1,2,3). Spin susceptibility is small at both small 
and large Q. This is quite different from the situation 
in a normal two-dimensional electron system, where spin 
susceptibility at small Q is large. 




x„ (Q. 0) 



Q(Qi) 



1 -0.5 0.5 1 

Qx(Qi) 



FIG. 3. (Color online) (a) The largest eigenvalue of spin sus- 
ceptibility tensor, Xg(QiO)i function of the wave- vector 
Q — (Q,0) along x-direction. Note that Q is in unit of 
the nesting wavevector Qi = |Qi| = 2ko. The red full 
(green dashed) curve correspond to the case with Ef = 0.7Es 
{0.5Es) and temperature T = 4 K. The blue dotted curve 
corresponding to the case with chemical potential 0.7 Es and 
T — 100 K. The dot-dashed curves denote other two eigen- 
values of for the T — 100 K case. The parameters of 
Bi2Te3 [see Table I] are used.— (b) Xg(QiO) as function of 
Q for Ef = 0.7^^, ksT = 1.3 x 10"^^^. x°G is in unit of 
0.05{k^JEs). 



e. Interaction correction Then by including the 
Hubbard interaction within the RPA approximation, one 
gets 



Xg(Qi,0) = 



X°g(Qi,0) 



l-aGUx%{Qi,0)- 



(34) 



Giy,Giz < 0. Hence the helical spin rotation axis Here 



XG(Qi,0) = i / di([aG(Qi,0,(^G(Qi,0))t])e'*(o+'"^). 
Jo 

(35) 
(36) 



1 



QG = 



1 + 



2 1 2 

ImI 



1 



< AG < 1. 



Here the awkward factor ao is due to the factor that 
the spin density (Tg(Qi) is not properly normalized. The 
normalized spin density is 



'5-g(Qi) = VaG^G*if,crtJ.{Qi 



(37) 



It is more obvious to see this through two specific cases: 
i) If {Gi^} is a real vector, e.g., (1,0,0), then ac = 1- 
This is the non-helical case. The normalized spin density 
is just CTG = cTz- n) If {Gi^} = -^(1,1,0), then ac = ^, 
which is the helical case. The normalized spin density is 
then a-Q = a-- = \{crx — ic^a)- The expectation values 
of both Uz and cr_ are less than or equal to unity, which 
signals the normalization. 

Accordingly, hereafter we use the normalized spin sus- 
ceptibility function. 



XG = acXG- 



One then gets 



Xg(Qi,0) 



X°g(Qi.O) 

-c/x^(Qi,o)' 



(38) 



(39) 
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which indicates the SDW instabiUty at low temperature. 

/. X other nesting wave-vectors From the symme- 
try of the system, the spin susceptibihty at other nesting 
wave-vectors can be obtained. Due to the C3 symmetry, 
X(Q2,0) = r^xiQi,0)'P, and x(Q3,0) = 7'tx(Q2,0)7' 
with V being the transformation matrix for rotation 
around the z-axis by 27r/3, i.e., the C3 operator. The 
spin density operators. 



(40) 



transform as (0 denotes time-reversal operation) 
C3 : a-G(QO ^ciQi+i) 

e : aG{Q^) ^ -[^g(Q»)]^ 

The complex vectors {Gi^} transform as 



(41) 



(42) 



IV. MEAN FIELD THEORY OF THE HELICAL 
SDW STATE 

A. Edges of 2D TIs 

The mean spin density in the helical SDW state is 

(ax) = 25o cos{Qz), (ay) = -25o sin(Qz), 

where 5o > (by properly choosing the coordinate) is 
the amplitude. It is noted that only the helical SDW 
with negative helicity (along z-axis) exists. This is due 
to the unique property of spin-momentum locking in TIs. 
If vq in Eq. (H]) is negative, then only the helical SDW 
with positive helicity exists. 

The properties of the ground state and quasi-particles 
in the helical SDW state can be explored via the mean 
field theory. Ignoring unimportant terms, the mean field 
Hubbard interaction is 

Hu = -2U[a+i-Q){a.m + (a+(-Q))a_(Q) 

-(a+(-g))(a_(g))], (43) 



with (cr^iQ)) : 
tonian is then 



((T+(— Q)) 5*0. The mean field Hamil- 




1 2 3 4 5 
DOS ( a.u. ) 



FIG. 4. (Color online) (a) Spectrum ±Ep (full curves) and 
spin configuration of quasi-particle in the helical SDW state 
compared with the initial spectrum ^p_|_ and Cp+Qt (dotted 
curves) and spin configuration, (b) Density of states of quasi- 
particle in the helical SDW state. 



with 



V2 



\ 



(pi 



--, Vp 



V2 



\ 



(pi 



e,i+B^ 



the Hamiltonian is diagonalized to be 

Hmf — 



(46) 



E Ep{rjlrjp-XlXp)+2USl (47) 

kp<p<kp 



with En = 



52. 



I . A gap is opened at the Fermi 

surface and the system becomes an insulator. Via the 
variational method, one obtains the gap equation, 



1 

U 



d 



[npi-Ep] - npiEp)] 



(48) 

which determines Sq. 

The quasi-particle in the helical SDW phase are de- 
scribed by rip and \p with —kF<p<kF- The spectrum 
and density of states are plotted in Fig. 4. It is indi- 
cated that the gap opens at the Fermi surface. The spin 
states of quasi-particles are no longer pure spin-up or 
spin-down, but mixed spin state, enabling back scatter- 
ing. 



B. Surfaces of 3D TIs with Hexagonal Fermi 
Surface 



Hmf = 

where B — —2USq. p = k + kp and ^pj, — —vqp — 
— ^p+Q-|-. The summation is restricted in the region 
-kp < p < kp as -2kp < k < and < fc Q < 2kp. 
Introducing a Bogoliubov transformation, 

rjp = UpCpi - VpCp+Q^, Xp = VpCpi -F UpCp+q^, (45) 



We first need to determine which spin density wave 
emerges in the SDW state. Following the Ginzburg- 
Landau theory, the free-energy functional can be written 
as 

Fgl =F2+ F^h (49) 

with 

F2 = \Y.XZ'MMM^)?- (50) 
^,1 
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Here Xiy,^ is the diagonalized and "normalized" [see 
Eq. ([55)) ] spin susceptibility tensor, (p^iq) is the SDW. 

denotes the free-energy to the fourth and higher or- 
ders of (j). At sufficient low temperature, x and F2 be- 
come negative, which drives the system into the SDW 
state. As 



(51) 



the SDW with largest xli, will first emerge as tem- 
perature is lowered down. According to the discus- 
sion in Sec. Ill B, such spin susceptibility is Xg(Q«''^) 
[i — (1,2,3)]. Therefore, the first emergent SDW is the 
helical SDW. As Xg(Q»' 0) is much larger than the others, 
there will be a temperature region, where only the helical 
SDW emerges. The following discussion is restricted in 
this temperature region;^ 

Free energy F2 of the concerned helical SDWs is. 



= JE^g'(Q-o)I'^« 



with 



(52) 



(53) 



being complex. According to Eq. (|4T|) . Fql should be 
invariant under the symmetry operations: 



(54) 



The C^y symmetry also gives another invariant opera- 
tion, 

(f)i-i. (55) 

Besides, Fgl respects the translational symmetry, 

r^r + d:(j),^(j),e'^^-'^. (56) 

Therefore, the symmetry allowed terms in Fql at the 
fourth order are, 



F4 = ^U4i\(j)i\^ -I- W42|0ip|0i+l|^ 



(57) 



To the sixth order, 

i 

+u,^[{cp,c^2hf + m*2r3f], (58) 

where the last term appears due to the translational sys- 
mmetry and Qi = 03I Those higher order terms de- 
termines whether the three SDWs coexist or only one of 
the SDWs is allowed (the stripe phase). Explicitly, af- 
ter reorganization, there are terms like (|(/>ip — |(/)i+ip)^" 
with n > 1, which prefer the difference |(|(/>ip — |0i+ip)|. 
If those terms are prominent, the stripe phase is favored. 



Below, we explore a mean field theory to describe the 
helical SDW states on the surfaces of 3D TIs. For sim- 
plicity, we restrict ourself to the stripe phase, where only 
one helical SDW exists. The mean field Hubbard Hamil- 
tonian is 



+ (^g(Q.))^ 



(59) 

The magnitude is determined by minimizing the 
Ginzburg-Landau free energy, whereas phase fluctuation 
of correspond to the gapless the spin wave excitations 
(Goldstone modes). In the stripe phase, only one of the 
three is nonzero. The mean field Hamiltonian is 



MF 



E' 



U 



-'k+Q,+ 




(60) 



with 



g,(k) = (7.+ (k)|aG(Q.)|"+(k + Q,)), = -—0*. 

(61) 

The summation is restricted in certain region as k 
and k -|- Qj dictate the same Hamiltonian. For example, 
for the case where only (pi is nonzero, k is restricted in 
the region where — Qi < fc^; < as Qi = (Qi,0). A 
significant feature of the Hamiltonian is that both the 
diagonal and off-diagonal terms are k-dependent, which 
may induce nontrivial Berry curvature in some cases. But 
we will not discuss this important property here. The 
mean field Hamiltonian is diagonalized as 



k.i 



via the following Bogoliubov transformation, 

?7ik = Uike~"''*Ck+ - WikCk+Q,+ 



with 



A, 



Uik 



Wik = 



Wike 



Ck+ + UikCk+Q;+, 



(62) 



(63) 




Cii< 



V21 



(64) 



and i/'ik = Arg[Si.gi(k)], Cik = 5(fk+ - <^k+Q.+)- The 
energy spectrum of the quasi-particle excitation is 



i?,k± = ±VCil + \B^9^{W + 2 (ek+ + a+Q. + ). (65) 

It should be mentioned that although the energy gap is 
k-dependent, it does not close at any k. 
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V. HELICAL MAGNETIC ORDER IN THE 
MAGNETICALLY DOPED SURFACES OF 3D TIS 

The magnetic order in the magnetically doped surfaces 
of 3D TIs has attracted a lot of interest)^"— Theoreti- 
cal investigations^ has shown that the Ruderman-Kittel- 
Kasuya-Yosida (RKKY) interaction is always ferromag- 
netic when the Fermi energy is close to the Dirac point. 
When a ferromagnetic order emerges, a gap is opened 
around the Dirac point. Recent experiments confirmed 
such results in Mn or Fe doped Bi2Se3 and Bi2Te3.— 

Here we consider the situation when the Fermi energy 
is much higher and the Fermi surface is hexagonal (and 
hence nested). From the discussion in previous sections, 
we know that the spin susceptibility function is peaked at 
the nesting wavevector where it is helical. Physically, the 
effective interaction between two magnetic impurities are 
mediated by the carrier spin density excited by one of the 
impurity and feeled by another. The spin susceptibility 
function describes such excitation. The nature of the 
spin susceptibility thus has profound implication on the 
effective interaction and the magnetic order. 

The above picture can be described well by the 
mean field Zener theory, which has been shown to be 
successful in the theory of dilute III(Mn)-V magnetic 
semiconductors.— The Hamiltonian of the system is 



STI 



I ■ Si 



(66) 



where Hsti is the Hamiltonian of carriers in the surfaces 
of 3D TIs. The last term describes the carrier-magnetic- 
impurity exchange interaction. / and i label magnetic 
impurities and carriers respectively (with spin S/ and s.^ 
respectively). 

In Zener theory, the equilibrium mean carrier and 
magnetic-impurity spin densities are calculated by min- 
imizing the Ginzburg-Landau free energy. Under the 
mean field approximation and neglect higher order cor- 
relations, the free energy functional is^ 



FGL[(f>] = -nMhsThi 



eM-^j/LkBT) 
JcjjSBsiSJclj/LkBT), 



(67) 



where (f) — i/X^u is the mean carrier spin density. 



L is the width of surface channel. The first term in right 
hand side is the free energy of magnetic impurities with 
S being the total spin quantum number of a single im- 
purity, um denotes the density of magnetic impurities. 
The second and third terms are the free energy of the car- 
rier system. The last term is the energy of carrier spin 
density under the mean field of the exchange interaction, 
with Bs being the Brillouin function. 

The underlying physics is that, the increase in car- 
rier spin density (jj reduces the first and last terms in 
the Ginzburg-Landau free energy, whereas it costs by 



increasing Fc[(j)] as the carrier system is in the param- 
agnetic state. The equilibrium value of 4> is determined 
by minimizing Fql [0] • At small carrier spin density (the 
"linear susceptibility regime" ) , Fc [(j)] can be written as 



(68) 



where x,y^ is the diagonalized and "normalized" [see 
Eq. (|38l) ] spin susceptibility. It is noted that the spin 
density 0j/(q) corresponding to the largest X:y„{q) is en- 
ergetically favored as it minimizes i^c[</']- In previous sec- 
tions, we have shown that the largest spin susceptibility 
is achieved at xciQi)- [see Fig. 3], where the correspond- 
ing 0G(Qi) is a helical spin density wave. Therefore, the 
energetically favored magnetic order is the helical mag- 
netic order, when Fermi surface is nested. 

It should be pointed out that the above equations can 
also be used to obtain the Curie temperature of the mag- 
netic orderiS2ii£ 



kBTc = 



SiS+^)J^nMXG{Q^,0) 



(69) 



The mean carrier spin density is a stripe helical one at the 
nesting wave- vector Q.; . The spins of magnetic impurities 
are aligned parallel or anti-parallel to the carrier spin 
density at their local positions, depending on the sign 
of the exchange constant J. Explicitly, the mean carrier 
spin density is 



(a^(r))=^(a,(Q.))e'^-- + 



where 



G 



2/i 



(70) 



(71) 



with /i = {x,y,z) and 0^ — (d'aiQi)) is complex. Here 
ac is defined in Eq. and Gi^ is defined in Eq. PO)) 
[the special case i = 1 is given approximately in Eq. p2p ] . 

The amplitude 4> = is determined by minimizing 
-Fgl[0]- There are three energetically favored configura- 
tions corresponding to the three Q^. For each configu- 
ration, the magnetic anisotropy is expected to be very 
large, as the spin susceptibility tensor x{Qi,0) is highly 
anisotropic [it has an eigen-value much larger than the 
other two, see Fig. 2(b)]. 

According to our calculation, for Mn doped Bi2Te3 sur- 
face states with Ep — 0.7Es (0.2 eV, corresponding elec- 
tron density 4x 10^^ cm"^), xg — 0.3*10"^ A'^eV"^ [see 
Fig. 3(a)], J ~ 10^ eV k^r^ the width of surface channel 
L = 50 A (inferred from Ref.[i3), riM = 2 x lO'^ A-^{~ 
0.1 mole fraction), S — 5/2, we get Tq — 30 K, which is 
not very low. 

It should be pointed out that further investigations on 
the problem are demanded. On one hand, the carrier- 
magnetic-impurity exchange interaction, which leads to 
carrier spin relaxation and shortens the propagating dis- 
tance of the carrier SDW excited by magnetic impurities. 
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should be included in the calculation of the spin suscep- 
tibility function. On the other hand, the exchange and 
correlation corrections to the spin susceptibility function 
should be included;^ A density-functional calculation 
will be appreciated f22i Via such improvement, the mag- 
netization can be calculated at arbitrary temperature, 
carrier and magnetic-impurity densities. We believe that 
the helical magnetic order is still favored after those cor- 
rections are included, according to the symmetry of the 
system. We assumed that the electron density can be 
tuned either by gate- voltage or doping by other dopants 
besides ferromagnetic impurities, which is in principle 
achievable in experiments 

Finally, we note that very recently. Ye et al. found 
that the helical magnetic order emerges in a chain of 
impurity spins in the surfaces of 3D TIs."*^ And Zhu et 
al. discussed the RKKY interaction when Fermi surface 
is hexagonal4^ 

VI. CONCLUSION 

We study spin susceptibility and magnetic order at the 
edges/surfaces of topological insulators when the Fermi 
surface is nested. We find that due to spin-momentum 
locking as well as time-reversal symmetry, spin suscepti- 
bility at the nesting wavevector has a strong helical fea- 
ture. It follows then, a helical SDW state emerges at 
low temperature due to Fermi surface nesting. The heli- 
cal feature of spin susceptibility also has profound impact 



^ C. L. Kane and E. J. Mele, Phys. Rev. Lett. 95, 146802 
(2005). 

^ B. A. Bernevig, T. A. Hughes, and S.-C. Zhang, Science 

314, 1757 (2006). 
^ L. Fu, C. L. Kane, and E. J. Mele, Phys. Rev. Lett. 98, 

106803 (2007). 

" L. Fu and C. L. Kane, Phys. Rev. B 76, 045302 (2007). 
^ J. E. Moore and L. Balents, Phys. Rev. B 75, 121306(R) 
(2007). 

^ R. Roy, Phys. Rev. B 79, 195322 (2009). 
M. Konig, S. Wiedmann, C. Briine, A. Roth, H. Buhmann, 
L. W. Molenkamp, X. L. Qi, and S.-C. Zhang, Science 318, 
766 (2007). 

* D. Hsieh, D. Qian, L. Wray, Y. Xia, Y. S. Hor, R. J. Cava, 

and M. Z. Hasan, Nature (London) 452, 970 (2008). 
^ Y. Xia, D. Qian, D. Hsieh, L. Wray, A. Pal, H. Lin, A. 

Bansil, D. Grauer, Y. S. Hor, R. J. Cava, and M. Z. Hasan, 

Nat. Phys. 5, 398 (2009). 
1° Y. L. Chen, J. G. Analytis, J.-H. Chu, Z. K. Liu, S.-K. Mo, 

X. L. Qi, H. J. Zhang, D. H. Lu, X. Dai, Z. Fang, S.-C. 

Zhang, I. R. Fisher, Z. Hussain, and Z.-X. Shen, Science 

325, 178 (2009). 
" C. Wu, B. A. Bernevig, and S.-C. Zhang, Phys. Rev. Lett. 

96, 106401 (2006). 
^2 H. Zhang, C. X. Liu, X. L. Qi, X. Dai, Z. Fang, and S.C. 

Zhang, Nat. Phys. 5, 438 (2009). 



on the magnetic order in the magnetically doped surfaces 
of 3D TIs. From the Zener theory, to the lowest order, 
we predicted a helical magnetic order in such system. 

The helical SDW order can be probed/determined ei- 
ther directly by spin resolved STM or indirectly by the 
existence of an energy gap at the Fermi surface via, e.g., 
ARPES. For spin pump-probe measurements, if a local 
spin density is excited, it will propagator with certain 
helicity along the nesting wavevectors. The helical mag- 
netic order in magnetically doped surface of 3D TIs can 
also be probed directly by spin resolved STM and other 
magnetic response measurements. 

Finally, recent studies indicate that topological insu- 
lators can also be achieved in cold-atom systems."'^ The 
tunability of inter-particle interaction (and many other 
properties) in such systems may be utilized to enhance 
the helical SDW order predicted in this paper. 
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